Is this word borrowed? An automatic approach to quantify the likeliness of borrowing in social media

Code-mixing or code-switching refer to the phenomenon of effortless and natural switching between two or more languages in a single conversation, sometimes even in a single utterance, by multilingual speakers. However, use a foreign word in a language does not necessarily mean that the speaker is code-switching, because often languages borrow lexical items from other languages. If a word is borrowed, it becomes a part of the lexicon of a language; whereas, during code-switching the speaker is aware that the conversation involves multiple languages and often the switching is intentional. Identifying whether a non-native word used by a bilingual speaker is due to borrowing or code-switching is not only of fundamental importance to theories of multilingualism, but it is also an essential prerequisite towards development of language and speech technologies for multilingual communities. In this paper, we present for the first time, a series of computational methods to identify the likeliness of a word being borrowed or code-mixed, based on the signals from social media. In particular, we use tweets from English-Hindi bilinguals from India to predict word borrowing. We first propose a method to sample a set of candidate words from the social media data using a context based clustering approach. Next, we propose three novel and similar metrics based on the usage of these words by the users in different tweets; we then apply these metrics to score and rank the candidate words indicating their likeliness of being borrowed. We compare these rankings with a ground truth ranking constructed through a human judgement experiment. The Spearman’s rank correlation between the two rankings (∼ 0.62 for all the three metric variants) is more than double the value (0.26) of the most competitive existing baseline reported in the literature. Some other striking observations are – (i) the correlation is higher for the ground truth data elicited from the younger participants (age < 30) than that from the older participants; since language change is brought about by the younger generation, this possibly indicates that social media is able to provide very early signals of borrowing, and (ii) those participants who use mixed-language for tweeting the least, provide the best signals of borrowing.